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Abstract. The current pace of technological innovation in web mapping offers 
new opportunities and creates new challenges for web cartographers. The 
continual development of new mapping applications and solutions produces a 
fundamental tension: the more flexible web mapping options become, the 
more difficult it is to maintain fluency in using and teaching these 
technologies. We address this tension by describing a case study process 
completed to meet the needs of the University of Wisconsin Madison 
Cartography program. Specifically, we conducted a sequence of three studies: 

(1) a competitive analysis study of contemporary web mapping technologies, 

(2) a needs assessment survey of internal designers/ developers regarding past 
experiences with these technologies, and (3) a diary study charting the 
implementation of a subset of potentially viable technologies for the program. 
The process successfully achieved the goal of identifying an appropriate suite of 
web mapping technologies for the UW program, but also revealed broader 
insights into web map design as well as ways to cope with evolving web 
mappi ng tech nol ogi es. 
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1. Introduction 

The current pace of innovation in Web Cartography is spectacular, with new 
releases of or substantial updates to web mapping technologies occurring 
almost daily (Haklay et al. 2008, Harrower 2008). However, the ever-evolving 
nature of technology results in a fundamental tension for cartographers. On 



one hand, the increasing flexibility and interoperability of web mapping 
technologies open new opportunities for web mappers; cartographers can do 
more now than ever. On the other hand, as technology evolves, so does the 
solution space from which cartographers can draw; it is increasingly difficult to 
establish and maintain one's bearings within this increasingly complex array of 
technologies. The research reported here addresses this technological tension 
by proposing a process— or streamlined, yet flexible sequences of stages for 
approach cartographic design (eg., Roth et al. 2009, Robinson et al. 2032)— to 
assess emergent web mapping technologies— or the compilation of 
framework, libraries, APIs, services, etc., that altogether enable the creation 
and dissemination of web maps (Kraak & Brown 2001, Peterson 2003). 

Our motivation for this work was two-fold. First, a refresh of the University of 
Wisconsin Madison (UW) Cartography lab exercises was needed in response 
to a broader shift in client-side web mapping away from standalone, 
proprietary technologies (e.g., Adobe Flash) and towards open technologies 
that leverage the HTML5/CSS3 web standards and J avaScript 1 . Second, the 
process served the purpose of building institutional knowledge on web 
mapping within the UW Cartography Laboratory, a research and technology 
center providing oncampus cartographic services and apprenticeship 
opportunities for students. Because of the dual goals, we designed the process 
to be generic, allowing for potential application by other university programs, 
government agencies, and cartography firms grappling with similar issues in 
technological change. Further, we designed the process for repeated 
application, allowing for continued maintenance of the UW program as web 
mappi ng technol ogy changes. 

The paper proceeds with four additional sections. I n the following section, core 
influences on the process from user-centered design are outlined. The three- 
stage, case study process is described in the third section, while the results 
from each stage are descri bed in the fourth section. The fifth and final section 
is reserved for cond uding remarks. 



1 At the time of this writing, web mapping design/ development is taught in the advanced UW 
Cartography courses only; introductory courses on static mapping using industry-standard 
commercial software (e.g., Adobe Creative Suite, ArcGI S) remain an integral component of the 
curriculum. Server-side technologies (e.g., PostGIS) are treated in a different GIS course and 
thus are not considered in this research, despite their acknowledged importance. All courses pair 
lab exercises with lectures drawing from theory and practice. 



2. Background 

The case study process for adapting to evolving web mapping technology was 
informed by scholarship from the field of Usability Engineering (Nielsen B93, 
Haklay 2010). Despite their title, usability engineers seek to identify the 
appropriate tradeoff between the usability (i.e., ease-of-use) and utility (i.e., 
usefulness) of an application, identifying missing or unneeded functionality 
from the system and critical issues in evoking this functionality for completing 
the users' goals (Fuhrmann et al. 2005, Robinson et al. 2011). Arguably, the 
best way to resolve this trade-off is to seek i nput from the targeted end users of 
the application, an approach described as user-centered design (Norman 
B88). User-centered design has been an active area of research within 
Cartography for over a decade (eg., Buttenfield B99, Gabbard et al. B99, 
Andrienko et al. 2002, Slocum et al. 2003, Fuhrmann et al. 2005, Robinson et 
al. 2005, Haklay et al. 2008) and increasingly is applied for the design and 
evaluation of web maps (eg., Haklay & Tobon 2003, Kramers 2008, Nivala et 
al. 2008, Roth & Harrower 2008, Goltekin et al. 2009, Roth et al. 2010). 

User-centered design relies on the elicitation of feedback on prototypes using 
archival or empirical methods. Many of these methods are qualitative and 
originate from social science, although there is a growing suite of methods 
specific to user-centered design (Marsh & Dykes 2008). Roth (2011a) provides 
a listing of common methods in Usability Engineering, organizing methods 
according to the source of feedback (Figure 1). Importantly, Nielson (B93) 
recommends a discount approach to user-centered design, recruiting a small 
number of participants (n=3-5) for each method to ascertain input and 
feedback quickly. Findings generated across multiple, discount evaluations can 
be triangulated to produce broader insights, an approach to 
design/ development described as convergent methods (Buttenfield 1999). 

Regarding web mapping technologies, the term user describes the 
designers/ developers of web maps as much as it does the audience of the 
resulting web maps. For the case study reported here, targeted users include 
senior undergraduate and graduate level students with at least one prior course 
in map design, GIS, and programming. The process described in the following 
section uses a discount, yet convergent, approach for evaluating web mapping 
technologies to efficiently, yet reliably, collect information regarding the 
appropri ate choi ce of tech nol ogy for the case study context. 





Method 


Similar or Related Methods 


■sed 


heuristic evaluation 


rules of thumb 


Expert-bc 


conformity inspection 


feature inspection, consistency inspection, standards 
inspection, autdefine checklist 


cognitive walkthroughs 


pluralistic walkthroughs, prototyping, storyboarding, 
Wizard of Oz 


try- based 


scenario-based design 


personas, scenarios of use, use case, context of use, theatre 


secondary sources 


content analysis, competitive analysis 


Thee 


automated evaluation 


automated interaction logs, unmoderated 
user-based methods 










participant observation 


ethnographies, field observation, MILCs, journal/diary 
sessions, screenshot captures 




surveys 


questionnaires, entry/exit surveys, blind voting, cognitive 
workload assessment 




interviews 


structured interviews, semi-structured interviews, 
unstructured interviews, contextual inquiry 


base 


focus groups 


supportive evaluation, Delphi 


User- 


card sorting 


Q methodology, concept mapping, affinity diagramming, 
brainstorming 




talk/think aloud 


verbal protocol analysis, co-discovery study 




interaction study 


performance measurement, controlled experiments 



Figure 1 User-centered design methods organized by source of feedback. Expert- 
based methods describe evaluations by experienced consultants. Theory-based 
methods describe evaluations by designers/developers. User-based methods describe 
evaluations by the targeted user group. While elicitation of feedback from users is 
recommended (hence user-centered design), it may not be feasible to recruit users 
given the stage in design and project resources. Figure redrawn from Roth (2011a). 



3. Methods 



Following a user-centered approach, three studies were selected from Figure 1 
and administered in sequence: (1) a competitive analysis study, (2) a needs 
assessment survey, and (3) a diary study. The three-stage process was designed 
to narrow iteratively the complete set of technologies, ultimately identifying a 
technology solution suitable for the case study context (but not necessarily all 
web mapping contexts). The selected methods allow for the process to be 
completed by a small team of designers/ developers (n=3-5) within a two week 
exploratory period directly foil owing a request for proposals or project kickoff. 

3.1. Competitive Analysis Study 

A competitive analysis study first was conducted to collect and evaluate all 
available web mapping technologies. A competitive analysis is a theory- 
based method based on secondary sources that critically compares a suite of 
related applications according to their relative merits (Nielsen B92). 
Completing a new competitive analysis at the start of each project is essential, 
given the pace of technological change in web mapping. 

The project team collected links to websites (the secondary sources) for all 
J avaScript web mapping technologies over a two-week period in Spring 2032, 
making use of keyword searches, popular blogs, and social media to find the 
technologies. Over 30 open-source or pay-servicej avaScript technologies were 
identified in total 2 (Figure 2). Two project members then independently coded 
the technologies according to the available representation functionality for 
graphically encoding information and interaction functionality for building 
user interfaces to modify the representation (Roth 20Ub); the codes 
themselves were derived from primary topics covered in lecture. Several non- 
J avaScript technologies (eg., GeoMoose, OpenScales, Processing) are included 
in Figure2 for comparison in functionality toj avaScript technologies. 

Project team members were instructed to apply the representation and 
interaction codes based on the documentation included within or linked from 
the websites, without experimenting with the code itself. Such an approach at 
the f i rst stage i n the process fol I ows di scount, convergent user-centered desi gn . 
This approach is further justified by the targeted user group— students— who 
need clear and comprehensive documentation to learn new technologies. A 
four point ordinal scale was applied during coding: (1) supported, (2) known 
work-around, (3) requires hack, and (4) not possible. 



2 Over a dozen more have been collected in the year following the formal competitive analysis. 



3.2. Needs Assessment Survey 

Following the competitive analysis study, a needs assessment survey was 
administered across the UW System (26 campuses) to elicit past experiences 
with the collected technologies as well as currently unmet web mapping needs. 
A survey is a user-based method that requires participants to respond to 
predetermined questions (Suchan & Brewer 2000). The survey was included as 
the second step in the process to acquire feedback about the collected 
technologies from targeted users outside of the project team; the survey was 
administered on line foil owing the discount, convergent approach. 

Twenty-one UW System employees participated in the needs assessment 
survey. Participation was limited to employees who either design/ develop web 
maps or who supervise such design/ development. The question protocol was 
divided into four sections: (1) basic biographic information, (2) current use of 
the technologies identified in the competitive analysis, (3) aspects of 
technologies that should be considered when selecting a technology, and (4) 
overarching opinions on designing web maps and teaching web map design. 
The non- biographical portion of the survey included J2 questions in total, with 
four Likert scale questions (Figures 3 & 4) and eight free response questions. 
The needs assessment survey was designed to take ]5 mi nutes. 

3.3. Diary Study 

I nsights from the competitive analysis study and needs assessment survey were 
combined to identify a subset of four candidate technologies believed to be 
viable options that may support the case study context. A diary study is a 
variation of the user- based participant observation method that requires 
participants to self-observe as they complete an activity. The diary study 
provided the deep experience working with a specific technology needed to 
identify an appropriate solution, but does so in a discount, convergent manner 
by relying on prior methods to reduce the total number of technologies under 
consideration; the self-observation further restricts impact on project 
resources. 

Four students representative of the targeted user group were recruited to 
complete an example web mapping scenario using one of the four assigned 
technologies (Figure 5); a fifth student completed the same scenario with all 
four technologies to improve reliability across participants (n=8 diaries). The 
scenario included 24 functional requirements in total, split between 
representation and interaction (Figure6). 



Participants were given a total of 40 hours to complete as many of the 
requirements as possible, mimicking constraints of an average work week. 
Students were not expected to complete all 24 requirements within the 
provided time period; the accomplished requirements therefore indicate the 
most straightforward functionality supported by the technology. Participants 
were required to log a diary entry every hour (40 entries total), with each entry 
including notes on the accomplished requirements in the past hour, key 
frustrations or breakthroughs, and their current satisfaction with the 
technology drawing from a provided list of emotions. Participants also were 
required to capture a screenshot of the web map and a version of their code 
with each diary entry (Haklay & Zafiri 2008). An exit survey was administered 
after reaching the 40 hour period to elicit additional feedback. 



4. Results and Discussion 
4.1. Competitive Analysis Study 

Results of the competitive analysis are illustrated in the Figure 2 matrix. 
Horizontally, the matrix reveals variation between specialist technologies 
designed to support specific functions (e.g., CloudMade Editor, Mapnik) and 
multi-purpose technologies supporting numerous functions (eg., D3, Google 
Maps, Leaflet, OpenLayers). Individual technologies generally fall into one of 
the following categories: (1) frameworks (eg., GeoMoose, MapServer), (2) 
open libraries (eg., D3, Modest Maps, Leaflet, OpenLayers, Polymaps, 
Raphael), (3) closed APIs (e.g., Bing Maps, Google Maps, MapQuest), and (4) 
tile rendering services (eg., Cloudmade, Mapnik, TillMill, TileStache). Given 
the curricular goals of teaching J avaScript specifically along with web map 
design generally, an emphasis was placed on open libraries that can be 
combined with other JavaScript libraries (e.g., jQuery); an exercise in tile 
rendering will be added in the future to an advanced graphic design course in 
Cartography. 

Vertically, the matrix provides a snapshot of trends in contemporary web map 
design. Widely supported representation functionality includes loading of 
vector or imagery basemaps, custom vector overlays, and choropleth or 
proportional symbol thematic maps. Widely supported interaction 
functionality includes panning, zooming, overlay of context layers, and 
retrieval of details using an information window; these four interaction 
operators altogether form the growing slippy web map design convention, 
snowing that this convention may be driven by limitations in the existing 



technologies, rather than by the actual needs of web map users. The 
functionality with the greatest variation across technologies include basemap 
styling and tile rendering for the representation category— explained by 
inclusion of custom rendering services in the matrix— and dynamic 
reprojection for the i nteraction category— showi ng the growi ng dependence on 
tile basemaps usi ng cyl i nd ri cal proj ecti ons. 
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Figure 2. Results of the competitive analysis study. Collection and coding was 
completed in Spring 2012; therefore, the matrix is no longer complete nor accurate, 
although arguably it never can be given the speed of technological advancements in 
web mapping. 



4.2. Needs Assessment Survey 

For space, discussion of the needs assessment survey is limited to participant 
responses to the Likert scales; however, the free response questions provided 
valuable clarification needed to interpret the Likert scale ratings. UW System 
employees make use of only a subset of technologies identified through the 
competitive analysis (Figure 3). Only the Google Maps API was used by a 
majority of participants in the past year (n=ll), with OpenLayers (n=9), 
ArcGIS Server (n=8), and Adobe Flash (n=6) used in the past year by a large 
minority. Interestingly, participants had not heard of a majority of the 
technol ogi es, rei nforci ng the fast pace of technol ogi cal change i n web mappi ng. 
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Please rate your engagement with the following web map technologies 
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Figure 3. Results of the needs assessment survey: level of engagement with existing 
web map technologies. 
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B. Please rate the importance of the following practical considerations: 
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C, Please rate the importance of the following technical considerations: 
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Figure 4. Results of the needs assessment survey: (a) web map characteristics; (b) 
practical considerations; (c) technical considerations. 



The additional Likert scale questions identified aspects of web mapping 
technologies that must be considered when selecting an appropriate solution. 
Participants rated interactivity as the most essential characteristic of web 
maps, with animation listed as the least essential (Figure 4a). Participants 
rated maintenance/ stability as the most important practical consideration and 
were divided evenly on access/ cost, indicating a split in open source versus 
commerci al tech nol ogi es across the U W System ( F i gu r e 4b) . P arti ci pants rated 
platform dependency and browser compatibility as the most important 
technical considerations, with location awareness rated as least important 
(Figure4c); this latter ranking provides an indication that many centers in the 
UW System have yet to makethejump to mobile at the time of testing. 

4.3. Diary Study 

I nsight from the competitive analysis and needs assessment were triangulated 
to identify four candidated technologies: three open libraries (D3, Leaflet, and 
OpenLayers) and one closed API (Google Maps) (Figure 5). Google Maps was 
included due to its frequent use across the UW System and for evaluation of 
different learning outcomes between open libraries and closed APIs. 
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Figure 5. Example solutions for the energy web mapping scenario resulting from the 
diary study: (red) D3; (blue) Google Maps; (green) Leaflet; (purple) OpenLayers. 



A. Accomplished Requirements by Technology 
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Figure 6. Results of the diary study: (a) total frequency of accomplished 
requirements by technology; (b) frequency of individual accomplished requirements; 
(c) participant experience by technology, according to reported emotional 
descriptions. 



Figure6 presents an overview of the diary study results. Again, each candidate 
technology has a pooled sample size of two, resulting in a maximum of 48 
requirements per technology (2x24). On average, participants completed the 
most scenario requirements using the Google Maps API (n=31), with Leaflet a 
close second (n=29); fewer requirements were accomplished with D3 (n=22) 
and Open Layers (n=21) (Figure 6a). There was substantial variation in the 
final maps by individual requirement, with the choropleth map and dynamic 
classification the only features implemented at least once using all four 
technologies; the interaction functions calculate, filter, and search were not 
implemented in any web map (Figure 6b). Interestingly, participant 
experiences with Leaflet were deemed more satisfying and less frustrating than 
the others, despite the Google Maps API resulting in the greatest number of 
implemented requirements (Figure 6c). Descriptions in the diaries and exit 
survey suggested that the open nature of the Leaflet code library, as well as its 
clear and comprehensive documentation, made the initial learning of Leaflet 
easier than the other alternatives. 



5. Conclusion 

The research presented here describes the use of a process for keeping pace 
with emerging web mapping technologies. Drawing upon key concepts in user- 
centered design, the process supports the identification of an appropriate 
technological solution— or combination of solutions— for a specific web 
mapping context. The purpose of the process is not to find an overall winner 
for all web mapping contexts, but to remain malleable as web mapping 
technology changes. Further, the process is designed to be completed by a 
small development team within a two week period and makes use of a 
discount, convergent approach to make efficient use of project resources. 

As a result of the process, we began using Leaflet in Fall 2032 as the base 
J avaScript library for the UW Cartography program, providing advanced labs 
that use pieces of D3 and the Google Maps API oncej avaScript is learned. The 
lab instructions and source code resulting from this project are available for 
download on Github: http://www.qithub.com/reroth/ q575-20B . The process 
will be administered regularly through the UW Cartography Lab— with the 
laboratory curriculum revised accordingly— to ensure that the UW-Madison 
Cartography program continues to evolve along with emergent web mapping 
technologies. 
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